The present invention relates to distributed systems, and more specifically, to a data processing method and apparatus for distributed systems.
HDFS (Hadoop Distributed File System) is a typical example of a distributed file system. The characteristics and drawbacks of an existing distributed file system are explained below using HDFS as an example.
The HDFS adopts the Master/Slave architecture. An HDFS cluster comprises a name node and a plurality of data nodes. The name node is a master storage node that manages the name space of the entire file system and an access request for a client. At the name node, operations such as opening/closing/renaming a file or catalog can be performed. The data node is a slave storage node that receives a read/write request from the client and performs creation, deletion and replication of a file block based on an instruction from the name node. In order to improve the reliability of storage, a file is generally stored with three replications, and each replication is stored with a fixed block size. An HDFS default data block size is 64 megabytes (MB). According to a replication mechanism of HDFS, segmented blocks are copied to other data nodes respectively based on the number of file replication factors, and then the name node records and saves which data blocks are included in one file replication as well as storage distribution information about being distributed in which data nodes (which is also called metadata information of the file system).
However, existing distributed file systems do not take the actual operation of a distributed program into consideration. Considering the MapReduce program as an example, when a data file stored with small data blocks is processed, more map tasks will be initiated, but relatively large overheads will also arise. When large blocks are adopted, the demands from the client and master server communication will be reduced, as will the space for the metadata which are needed to store by the master server, but available parallel computing resources cannot be utilized effectively. Referring to the example of Table 1, when 1 terabyte (TB) of data is sorted on 20 nodes, suppose each node can run 40 map tasks to the utmost simultaneously, then there are 800 map slots in total. It is seen that when adopting large blocks, only 200 map tasks are utilized in the second round, and the available computing resources cannot be utilized fully.
TABLE 1BlockTotal timeSizerunning condition of Map tasksconsumed 1 GBFirst round: using 800 map tasks to sort80 + 80 =800 GB data, taking 80 sec160 secSecond round: using 200 map tasks to sortthe remaining 200 GB data, taking 80 sec256 MBIn each round, use 800 map tasks to sort 20028 * 5 =GB data, taking 28 sec; 5 rounds in total140 sec
Another typical example of a distributed file system is GFS (Google File System). GFS and HDFS have a correspondence relationship in terminology, for example, a master node of GFS corresponds to the name node in HDFS, a GFS block server corresponds to a data node in HDFS, a block in GFS corresponds to a block in HDFS; GFS likewise has the above problems.
Therefore, the prior art still has room to improve, and it is desirable for a data processing method and apparatus for a distributed system.